Fast Intra- and Inter-Prediction Mode Decision in H.264 Advanced Video Coding

H.264/AVC, the latest video standard, adopts rate-distortion optimization (RDO) technique to obtain the best intraand interprediction, while maximizing visual quality and minimizing the required bitrate. However, the full RD cost calculation for all intra-prediction modes, the exhaustive searches for finding optimal motion vectors for all block sizes, and the multiple references frame procedure considerably increase its computational complexity with the allowed number of prediction modes. In order to reduce the complexity, here we propose a new approach for both interand intra-mode decisions, that takes into account the two effective parameters, image content type and the quantization parameter. The proposed fast intra-prediction mode selection strategy uses some observations on the interior and the exterior MB properties to select a subset of candidate modes at different quantization parameters. Also, the fast inter-prediction mode decision approach uses split/merge procedure based on correlation of motion vectors and motion details of video objects. Also, here we use a context-based adaptive method to speed up the multiple reference frames motion estimation that is based on interand intra-prediction residues and quantization parameters. As such, only a subset of interand intra-modes is chosen for RDO calculation. Experimental results show that the proposed algorithm, reduces the total encoding cost with negligible loss in PSNR and a slightly increase in the required bitrate, when compared to RDO and other fast algorithms reported in the literature.